Skip to content

Refactor: expose resolved judging function - #194

Draft
luca-belli wants to merge 3 commits into
docs/judge-command-planfrom
refactor/judge-runtime-entrypoint
Draft

Refactor: expose resolved judging function#194
luca-belli wants to merge 3 commits into
docs/judge-command-planfrom
refactor/judge-runtime-entrypoint

Conversation

@luca-belli

@luca-belli luca-belli commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Mirror of #190 for the judging side: give the judge domain an entry point that
takes fully resolved values, so vera judge can call it without inheriting a
script's argument conventions.

Behavior-preserving plumbing. Adds no CLI surface — vera judge itself is a
follow-up. Stacked on #193, which settles the design decisions this implements.

What changed

judge/run.py::run_judging (new, 125 lines) — keyword-only, no parameter
defaults. Receives resolved rubric paths, a resolved transcripts directory, and
an explicit output target; loads the rubric and conversations; calls the existing
judge_conversations. It parses no arguments, reads no manifest, applies no
defaults, picks no output location, and does not configure debug logging.

Placement differs from #190 deliberately. The generation side needed a
temporary root-level boundary (generate.main) because the permanent generate/
package does not exist yet. judge/ is already the final home, and
architecture.md has vera judge delegating to the judge package — so this goes
straight in, and vera_cli/judge.py will import it directly. One fewer
indirection than the generate side carries.

RubricConfig.from_paths — construction from three resolved paths. load
and load_bundle delegate to it, keeping manifest reading out of the
resolved-value path. Behavior unchanged.

load_manifest_rubric_paths in utils/rubric_manifest.py — resolves a
manifest's three rubric files relative to the manifest, beside the existing
load_manifest_personas and load_manifest_persona_context_template which
resolve other fields the same way. This was review feedback: the first pass put
this logic in judge.py, adding a third copy of the manifest-relative rule to a
file scheduled for deletion, when RubricConfig.load_bundle already applied it.
Now both share one implementation that outlives judge.py. (vera's target
resolution needs none of it — ResolvedTarget already carries these paths.)

judge.py::main is now visibly CLI glue. It keeps its namespace signature —
it is this script's own entry point, not a domain one, and run_pipeline.py
calls it unchanged — but everything script-specific is now isolated as CLI
policy: model shorthand parsing, the manifest-path input form, output-location
choice, resume validation, and set_debug. _resolve_output_target holds the
three-branch default and the resume checks. Single-conversation judging stays
here, marked legacy-only, since vera judge drops that mode per #193.

judge/runner.py is untouched, matching #190's "the existing large runner
does not grow."

Size

#190 (generate) this
Files 6 6
Diff +271 / −263 +308 / −89 (125 of it the new module)

Smaller in substance than the line count suggests: #190 had to create the
resolved-input application function by extraction, whereas judge_conversations
already took resolved values — this only adds the load-and-run layer above it.

Incidental win

14 Pyright errors in judge.py are gone (baseline 14 → 0). All came from
building judge_kwargs as an untyped dict and unpacking it into a typed
signature, which defeats checking entirely. Explicit keywords restore it. The 10
remaining errors in judge/rubric_config.py are pre-existing pandas
Series.__bool__ issues in code this PR does not touch.

Test changes

Five TestJudgeMain tests patched load_conversations, judge_conversations,
and RubricConfig as attributes of judge.py; those seams now live in
judge/run.py. Re-pointed, with no behavior assertion weakened.

The RubricConfig.load_bundle assertions became from_paths and are now
stronger: instead of confirming a manifest path was forwarded, they confirm
the three rubric files were resolved out of it
(data/SI/rubric_manifest.json → data/SI/rubric.tsv plus the two prompt
files). test_main_loads_distinct_rubric_bundles_end_to_end keeps its
real-parsing design and still proves --rubrics selects the bundle rather than
being a no-op.

Validation

  • Full non-live suite: 1,023 passed, 8 deselected — same count as the
    generate branch, so nothing regressed
  • Coverage: 74.90% (gate: 30%)
  • Ruff format/check: no new findings (3 pre-existing E501s in
    judge/response_models.py, untouched)
  • Pyright on judge.py, judge/run.py, utils/rubric_manifest.py: 0 errors,
    verified against a stashed baseline

🤖 Generated with Claude Code

luca-belli and others added 3 commits August 13, 2026 16:27
@luca-belli @claude
refactor: expose resolved judging function
0fbbec4 
Mirror of #190 for the judging side: give the judge domain an entry point
that takes fully resolved values, so `vera judge` can call it without
inheriting a script's argument conventions.

Add `judge/run.py::run_judging` -- keyword-only, no parameter defaults. It
receives resolved rubric paths, a resolved transcripts directory, and an
explicit output target; loads the rubric and conversations; and calls the
existing `judge_conversations`. It parses no arguments, reads no manifest,
applies no defaults, chooses no output location, and does not configure
debug logging.

Unlike the generation side, this goes straight into the permanent `judge/`
package rather than behind a temporary root-level boundary function: `judge/`
is already the final home, and architecture.md has `vera judge` delegating
to the judge package.

Add `RubricConfig.from_paths` for construction from three resolved paths;
`load` and `load_bundle` now delegate to it, so manifest reading stays out
of the resolved-value path. Behavior unchanged.

Reduce `judge.py::main` to CLI glue. It keeps its namespace signature -- it
is this script's own entry point, not a domain one, and `run_pipeline.py`
calls it unchanged -- but everything script-specific is now visibly CLI
policy: model shorthand parsing, manifest resolution, output-location
choice, resume validation, and `set_debug`. Two helpers isolate the policy,
`_resolve_rubric_paths` and `_resolve_output_target`. Single-conversation
judging stays here and is marked legacy-only, since `vera judge` drops it.

Incidentally clears 14 Pyright errors in `judge.py`, all from building
`judge_kwargs` as an untyped dict and unpacking it into a typed signature,
which defeated checking entirely. Explicit keywords restore it.

The judge runner itself is untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@luca-belli @claude
refactor: put manifest rubric-path resolution in one place
7795c08 
Review catch: the previous commit added a third copy of "rubric bundle
manifest paths are relative to the manifest's own folder" -- and put it in
`judge.py`, a file scheduled for deletion.

The helper itself is legacy-only, since `vera judge` resolves rubric paths
through target resolution and never sees a bare manifest path. Its logic is
not: `RubricConfig.load_bundle` already applied the same rule.

Move it to `utils/rubric_manifest.py` as `load_manifest_rubric_paths`,
beside the existing `load_manifest_personas` and
`load_manifest_persona_context_template`, which resolve other manifest
fields the same way. Its keys match `RubricConfig.from_paths`, so
`load_bundle` collapses to one line and both paths now share one
implementation that outlives `judge.py`.

The third copy, in `vera_cli/targets.py`, resolves a different manifest kind
and is left alone.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@luca-belli @claude
test: point judge CLI tests at the moved seams
a2476d9 
Loading and dispatch moved from `judge.py` into `judge.run.run_judging`, so
the five `TestJudgeMain` tests patched attributes that no longer exist on the
script module. Re-point them; no behavior assertion is weakened.

The four folder-path tests now patch `load_conversations`,
`judge_conversations`, and `RubricConfig` on `judge.run`, where they are used.
Everything those tests assert about argument forwarding, output-target
selection, and resume still holds end to end.

`RubricConfig.load_bundle` is no longer called by `main()` at all, since the
manifest is resolved to paths first, so those assertions become
`RubricConfig.from_paths`. This checks more than before: instead of confirming
a manifest path was forwarded, it confirms the three rubric files were
actually resolved out of it, e.g. `data/SI/rubric_manifest.json` ->
`data/SI/rubric.tsv` plus the two prompt files.

`test_main_loads_distinct_rubric_bundles_end_to_end` keeps its real-parsing
design, substituting the real `from_paths` for the real `load_bundle`, so it
still proves `--rubrics` selects the bundle rather than being a no-op.

The single-conversation test continues to patch `judge.py` attributes, because
that path genuinely still lives there as legacy-only code.

Full non-live suite: 1,023 passed, coverage 74.90%.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@luca-belli luca-belli mentioned this pull request Aug 14, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@luca-belli